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(54) ROBOT DEVICE AND ITS CONTROL METHOD 

(57)Abstract: 

PROBLEM TO BE SOLVED: To save labor for 
troublesome teaching work and calibration and 
allow easy work for objects, as predetermined. 
SOLUTION: In teaching work, a robot controller 

5 stores a position Pd of a gripper 3 in the state of 

' /" ~I <- : lfi ;7 : ^7 T S r ipping the object 1 1 and further stores any 

| f J^f^ / J / position (Ps) after the object 1 1 is released. In 

I S'*) q / / clamping work, the robot controller 5 makes the 

' ~ — . 0 gripper 3 follow the object 1 1 being carried, go to 

f I the object 1 1 in accordance with the positions Ps, 

r^Vh r t ^3 / - fr - Pd and grip the object 1 1 . 

L 3y ' v -T i^lzil- ' 



JP 2002-018754 

Date of Publication: 01/22/02 



CLAIMS 



[Claim(s)] 

[Claim 1]A robot device which grasps a subject which exercises relatively, comprising: 
A holding means which has a gripping mechanism which grasps said subject and said 
gripping mechanism comprised movable. 

An imaging means which is fixed movable with a gripping mechanism of said holding 
means, and picturizes a subject. 

A feature amount extracting means which extracts coordinates of one or more 
characteristic quantity from a picture of a subject which said imaging means picturized. 
A position of a gripping mechanism at a time of making said gripping mechanism grasp 
said subject is memorized as the 1st end point position, A position of said gripping 
mechanism when moving said gripping mechanism within limits which can make said 
subject able to open wide and in which said imaging means can picturize a subject is 
memorized as the 2nd end point position, A memory measure which memorizes 
coordinates of each characteristic quantity of said subject which said imaging means 
picturized in said 2nd end point position, and said feature amount extracting means 
extracted. 

A calculating means which computes a move procession for moving from said 2nd end 
point position to said end point position of 1 . 

A follow-up control means by which coordinates of each characteristic quantity of a 
subject under movement extracted by said feature amount extracting means carry out 
control which said gripping mechanism is made to follow to a subject under said 
movement by moving said holding means so that it may be in agreement with coordinates 
of each characteristic quantity memorized by said memory measure. 
A grasping control means which controls said holding means to bring said gripping 
mechanism close to a subject under said movement based on a move procession which 
said calculating means computed, and to make said subject grasp when a subject while 
said gripping mechanism is exercising is followed. 

[Claim 2]While said gripping mechanism separates said memory measure prescribed 
distance to a subject under said movement by said follow-up control means, when it 
follows, Coordinates of each characteristic quantity of said subject which said feature 
amount extracting means extracted are newly memorized, The robot device according to 
claim 1 which memorizes a position of said gripping mechanism when said follow-up 
control means makes said gripping mechanism follow a subject thoroughly from said 2nd 
end point position using coordinates of each newly memorized characteristic quantity as 
the 2nd new end point position. 

[Claim 3]In a control method of a robot device which grasps a subject which exercises 
relatively by a gripping mechanism constituted movable, A position of a gripping 
mechanism at a time of making said gripping mechanism grasp said subject is memorized 
as the 1st end point position, A position of said gripping mechanism when making said 
subject open wide and moving said gripping mechanism within limits which can picturize 
a subject is memorized as the 2nd end point position, In said 2nd end point position, 
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picturize said subject, and coordinates of one or more characteristic quantity are extracted 
from a picture of said picturized subject, In the state where memorized coordinates of 
each of said extracted characteristic quantity, computed a move procession for moving 
from said 2nd end point position to said end point position of 1, and it was fixed movable 
with said gripping mechanism. Picturize a subject under movement and coordinates of 
one or more characteristic quantity are extracted from a picture of a subject under said 
picturized movement, Coordinates of each characteristic quantity of a subject under said 
extracted movement by moving said gripping mechanism so that it may be in agreement 
with coordinates of each of said characteristic quantity memorized. When said gripping 
mechanism is made to follow a subject under said movement and a subject while said 
gripping mechanism is exercising is followed, based on said computed move procession, 
said gripping mechanism is brought close to a subject under said movement, A control 
method of a robot device of making said gripping mechanism grasping said subject. 
[Claim 4]While said gripping mechanism is made to follow a subject under said 
movement, and said gripping mechanism separates prescribed distance to a subject under 
said movement, when it follows, in the state where it was fixed movable with said 
gripping mechanism. Picturize a subject under movement and coordinates of one or more 
characteristic quantity are extracted from a picture of a subject under said picturized 
movement, Newly memorize coordinates of each characteristic quantity of said said 
extracted subject, and coordinates of each of said newly memorized characteristic 
quantity are used, A control method of the robot device according to claim 3 of 
memorizing a position of said gripping mechanism as the 2nd new end point position 
when said gripping mechanism is moved so that said gripping mechanism may follow a 
subject thoroughly from said 2nd end point position, and followed thoroughly. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to a robot device and a method for 
controlling the same, and relates to a robot device which grasps the object under 
movement especially, and a method for controlling the same. 
[0002] 

[Description of the Prior Art]When working to a subject using a robot (for example, 
objective grasping), information, including the position of a subject, a posture, etc., is 
acquired by a sensor, and many examples which work based on the sensor information 
are put in practical use. As such an equipment configuration, it divides roughly and, 
generally there are the following two methods. 

[0003 ]For example, there are some which detect the position of a subject and a posture 
from the picture acquired by picturizing with the camera which had the subject which has 
a conveyor top conveyed fixed, and transmit the information to a robot as indicated to 
JP,8-63214,A. This robot is working grasping etc. from the position information on a 
subject. 

[0004]Sensors, such as a camera, are attached to the hand of a robot, the target positional 
attitude is detected from the picture from a camera as mentioned above, and there are 
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some which follow the subject which moves based on the information, and work grasping 

etc. 

[0005] 

[Problem(s) to be Solved by the InventionJHowever, when realizing the work of grasping 
of a subject, etc. using the camera installed apart from the robot like the former, the work 
which asks for the installed position of a robot and a camera correctly beforehand is 
needed. It is necessary to perform the camera calibration which computes a focal 
distance, a lens strain coefficient, etc. of the camera beforehand. 
[0006]In addition, in the example in JP,8-63214,A etc., it is necessary to install so that 
the optic axis of a camera may generally become vertical still more nearly parallel to the 
transportation direction of a conveyor at a conveyor flat surface in the x axis or the y-axis 
in a screen. Since the subject is running by the band conveyor etc., in addition to the 
position attitude information of the subject detected by the picture from a camera, another 
means (for example, encoder of a conveyor) needs to detect the speed information of a 
conveyor, etc., and it is necessary to generate the command value to a robot based on 
both information. The orbit at the time of a robot grasping a subject from the installation 
position information of a camera and a robot, the position information on a subject, and 
the speed information of a conveyor must be generated. Therefore, there are problems, 
like the procedure for teaching work and trajectory generation becomes very complicated. 
[0007]It can say such a problem that the same may be said of the example of the latter 
which installed the camera to the hand of the robot in conventional technology. That is, 
when a subject is grasped by the gripper etc. which were attached to the hand of a robot 
based on the position attitude information of the subject detected from the picture from a 
camera, What is called a hand eye calibration that needs to change the position attitude 
information of the subject acquired from the camera into the position attitude information 
seen from the standard coordinates of the robot, and searches for the physical relationship 
of a robot hand and a camera precisely beforehand is needed. When sensors, such as a 
camera, are especially installed in a robot hand, a camera position relation with a robot 
hand frequently changes with the malfunctions under work, etc., and it is necessary to do 
calibration work again in that case. When the subject is moving like the former example, 
since the position attitude information of the subject seen from the robot normal 
coordinate changes every moment, it becomes very complicated [ the trajectory 
generation for grasping ]. 

[0008]In the former, even if the method of the former and the latter changes neither a 
component (the installation condition of a conveyor, a bearer rate, or the installed 
position relation between a camera and a robot), nor the shape of a subject which should 
be grasped, Whenever the robot and sensor to be used changed, the above-mentioned 
highly precise calibration and complicated teaching work needed to be performed. 
[0009]This invention is proposed in order to cancel the problem mentioned above, and it 
is a thing. 

The purpose is to provide a robot device which can save the time and effort which 
performs work and a calibration, and can make predetermined work do on a subject 
easily, and a method for controlling the same. 

[0010] 

[Means for Solving the Problem]A holding means from which the invention according to 
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claim 1 has a gripping mechanism which grasps a subject and which said gripping 
mechanism comprised movable, An imaging means which is fixed movable with a 
gripping mechanism of said holding means, and picturizes a subject, A feature amount 
extracting means which extracts coordinates of one or more characteristic quantity from a 
picture of a subject which said imaging means picturized, A position of a gripping 
mechanism at a time of making said gripping mechanism grasp said subject is memorized 
as the 1st end point position, A position of said gripping mechanism when moving said 
gripping mechanism within limits which can make said subject able to open wide and in 
which said imaging means can picturize a subject is memorized as the 2nd end point 
position, A memory measure which memorizes coordinates of each characteristic 
quantity of said subject which said imaging means picturized in said 2nd end point 
position, and said feature amount extracting means extracted, A calculating means which 
computes a move procession for moving from said 2nd end point position to said end 
point position of 1, Coordinates of each characteristic quantity of a subject under 
movement extracted by said feature amount extracting means by moving said holding 
means so that it may be in agreement with coordinates of each characteristic quantity 
memorized by said memory measure. When a follow-up control means which carries out 
control which said gripping mechanism is made to follow to a subject under said 
movement, and a subject while said gripping mechanism is exercising are followed, 
Based on a move procession which said calculating means computed, said gripping 
mechanism is brought close to a subject under said movement, and it has a grasping 
control means which controls said holding means to make said subject grasp. 
[001 l]The robot device according to claim 1 grasps a subject which exercises relatively 
to the robot device concerned. Therefore, a case where said robot device exercises and 
said subject is being fixed, when said robot device and said subject are exercising at a 
speed different, respectively, and not only when said robot device is fixed and said 
subject exercises, but when, it can apply. Although said subject may be carrying out the 
acceleration straight-line motion of linear uniform motion, the uniform circular motion, 
etc. relatively to said robot device, it may be standing it still relatively. 
[0012]First, the operator needs to perform the following two things as teaching work. A 
gripping mechanism is moved to the 1st end point position, and a subject is made 1st to 
grasp here. A subject is made to release from a gripping mechanism and an imaging 
means moves [ 2nd ] a gripping mechanism within limits which can picturize a subject to 
the 2nd end point position. Such teaching work is performed and a memory measure is 
made to memorize the 1st and 2nd end point positions. A memory measure is made to 
also memorize coordinates of one or more characteristic quantity extracted from a picture 
which an imaging means picturized in the 2nd end point position. An imaging means 
serves as a sensor of a gripping mechanism, and it is being fixed so that it may move with 
a holding means. A move procession for moving from the 2nd end point position to the 
1st end point position is computed. 

[0013]If such teaching work is completed, grip operations which grasp a subject under 
movement can be performed. A gripping mechanism is moved so that coordinates of each 
characteristic quantity of a subject under movement may be in agreement with 
coordinates of each characteristic quantity memorized by said memory measure. That is, 
what is called a visual servo is performed so that a subject while a gripping mechanism is 
exercising may be followed. Relative physical relationship of a subject at this time and a 
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gripping mechanism is the same as a subject at the time of teaching work, and that 
relative of a gripping mechanism. When a subject while said gripping mechanism is 
exercising is followed, based on the 1st move procession that said calculating means 
computed, said gripping mechanism is brought close to a subject under said movement, 
and said subject is made to grasp. 

[0014]It is desirable for a gripping mechanism to follow a subject under movement 
thoroughly in a case where a visual servo is being performed. However, a gripping 
mechanism may follow, separating prescribed distance to a subject under movement. In 
such a case, it is preferred that re-teaching work which is indicated to claim 2 is 
performed. While said gripping mechanism separates said memory measure prescribed 
distance to a subject under said movement by said follow-up control means, when it 
follows, specifically, Coordinates of each characteristic quantity of said subject which 
said feature amount extracting means extracted are newly memorized, What is necessary 
is just to memorize a position of said gripping mechanism when said follow-up control 
means makes said gripping mechanism follow a subject thoroughly from said 2nd end 
point position using coordinates of each newly memorized characteristic quantity as the 
2nd new end point position. 

[0015]The invention according to claim 3 is the control method of a robot device which 
grasps a subject by a gripping mechanism constituted movable, A position of a gripping 
mechanism at a time of making said gripping mechanism grasp said subject is memorized 
as the 1st end point position, A position of said gripping mechanism when making said 
subject open wide and moving said gripping mechanism within limits which can picturize 
a subject is memorized as the 2nd end point position, In said 2nd end point position, 
picturize said subject, and coordinates of one or more characteristic quantity are extracted 
from a picture of said picturized subject, In the state where memorized coordinates of 
each of said extracted characteristic quantity, computed a move procession for moving 
from said 2nd end point position to said end point position of 1 , and it was fixed movable 
with said gripping mechanism. Picturize a subject under movement and coordinates of 
one or more characteristic quantity are extracted from a picture of a subject under said 
picturized movement, Coordinates of each characteristic quantity of a subject under said 
extracted movement by moving said gripping mechanism so that it may be in agreement 
with coordinates of each of said characteristic quantity memorized. When said gripping 
mechanism is made to follow a subject under said movement and a subject while said 
gripping mechanism is exercising is followed, based on said computed move procession, 
said gripping mechanism is brought close to a subject under said movement, Said 
gripping mechanism is made to grasp said subject. 

[0016]This invention is a method for controlling a robot device indicated to claim 1, and 
the operator should just perform two things mentioned above as teaching work. That is, 
only by an operator performing two easy teaching work, a gripping mechanism can be 
made to be able to follow a subject which is exercising and the subject can be grasped. 
[0017]It is desirable for a gripping mechanism to follow a subject under movement 
thoroughly in a case where a visual servo is being performed. However, a gripping 
mechanism may follow, separating prescribed distance to a subject under movement. In 
such a case, it is preferred that re-teaching work which is indicated to claim 4 is 
performed. While said gripping mechanism specifically separates prescribed distance to a 
subject under said movement, when it makes said gripping mechanism follow a subject 
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under said movement, and it follows, it is in a state fixed movable with said gripping 
mechanism, Picturize a subject under movement and coordinates of one or more 
characteristic quantity are extracted from a picture of a subject under said picturized 
movement, Newly memorize coordinates of each characteristic quantity of said said 
extracted subject, and coordinates of each of said newly memorized characteristic 
quantity are used, What is necessary is just to memorize a position of said gripping 
mechanism as the 2nd new end point position, when said gripping mechanism is moved 
so that said gripping mechanism may follow a subject thoroughly from said 2nd end point 
position, and followed thoroughly. 
[0018] 

[Embodiment of the Invention] Hereafter, it explains in detail, referring to drawings for an 
embodiment of the invention. 

[0019](A 1st embodiment) As shown in drawing...]., the robot device concerning this 
embodiment grasps the subject 1 1 currently conveyed by conveyor 10 in the state where 
it was fixed. This invention is not limited to such an embodiment, and if the subject 1 1 is 
exercising relatively to a robot device, it is applicable. That is, they may be a case where 
a robot device exercises and the subject 1 1 is standing it still, and a case where it is 
exercising at the speed from which a robot device and the subject 1 1 differ. 2nd and 3rd 
embodiments are also the same. 
[0020]A robot device is provided with the following. 
CCD camera 1 which picturizes the subject 1 1 . 

The sensor controller 2 which extracts characteristic quantity from the picture which 

CCD camera 1 picturized. 

The gripper 3 which grasps the subject 1 1 . 

The robot controller 5 which performs the robot arm 4 which moves the gripper 3 in the 
direction of a three dimension, coordinate conversion, trajectory generation, etc., has a 
servo circuit, amplifier, etc. which are not illustrated, and controls operation of the 
gripper 3 or the robot arm 4. 

[0021]CCD camera 1 has adhered near the gripper 3 which is a tip of the robot arm 4. 
The robot controller 5 computes error vector E (t) and the speed commanding value 
vector V (t) based on the image characteristic quantity Fi obtained with the sensor 
controller 2, and controls movement of the robot arm 4, and grasping of the subject 1 1 by 
the gripper 3 according to these values. The conveyor 10 conveys the subject 1 1 at a 
fixed speed. 

[0022]Before the robot device of such composition actually grasps the subject 1 1 
currently conveyed, teaching work for grasping the subject 1 1 is performed. 
[0023](Teaching work) First, an operator makes a robot device grasp the subject 11, and 
teaches the robot end point position P. The position which should be grasped may be 
restricted by the shape of gripping mechanisms, such as the gripper 3, and the subject 1 1 . 
The position which should be grasped for next processing (for example, attach other parts 
to the subject 1 1, or supply to a parts box) may be specified beforehand. 
[0024]Here, as shown, for example in drawing 2 , the subject 1 1 was formed in 7 
prismatic forms, and is provided with the seven sides 111-117. Six attachment holes (for 
example, tapped hole) for attaching the parts which are not illustrated are formed in the 
upper surface 1 18 of the subject 11. Such an attachment hole is not formed in the 
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undersurface 1 19 of the subject 1 1 . As for the portion by which the subject 1 1 is grasped, 
it is preferred that they are the side 1 1 1,1 15 in which it faces among seven sides in 
consideration of stability when grasped, or the side 1 13,1 16. Although the subject 1 1 of 
the above-mentioned shape was used in this embodiment, this invention may not be 
limited to this shape and the subjects 1 1 may be other shape. And teaching work is 
specifically performed according to processing of step ST1 to step ST6 shown in drawing 
3. 

[0025]In step ST1, an operator shifts to step ST2, as the gripper 3 grasps the side 111,115 
or the side 113,116 of the subject 1 1 using the teaching pendant etc. which are not 
illustrated. 

[0026]In step ST2, as shown in drawing 4, the gripper 3 is in the state which grasped the 
subject 11, and the robot controller 5 memorizes, the position, i.e., robot end point 
position Pd, of the gripper 3 in robot standard coordinates, and shifts to step ST3. The 
"position" said here is expressed with the parameter of 6 flexibility including a posture. 
The parameter of 6 flexibility including a posture is called "position" like the following. 
[0027]In step ST3, the robot controller 5 makes the subject 1 1 which the gripper 3 
grasped release, and moves the gripper 3 to arbitrary positions, and shifts to step ST4. 
[0028]It requires that the position which the gripper 3 moves is in the visual field range 
of CCD camera 1 . That is, it is a position where the characteristic quantity of the subject 
1 1 is extracted from the picture picturized with CCD camera 1 . It is a position where 
characteristic quantity which should be extracted, such as a position to which the subject 
1 1 is not settled in the view of CCD camera 1, and a position in which CCD camera 1 
picturizes the subject 1 1 from just beside, is specifically suitably obtained from a picture. 
[0029]In step ST4, the robot controller 5 memorizes, as shown in drawing 5 , the position 
Ps, i.e., the robot end point position, of the gripper 3 in the present robot standard 
coordinates, and it shifts to step ST5. 

[0030]Here, the following formulas will be realized if the positional attitude 
transformation matrix for changing into Pd from the robot end point position Ps is made 
into Po- 

[0031]Ps-Po=PdthereforePo=Ps 1 and the Pd robot controller 5 compute positional 
attitude transformation-matrix Po at this time, and memorizes this. The robot controller 5 
may compute positional attitude transformation-matrix P 0 in the case of the grip 
operations mentioned later. 

[0032]In step ST5, CCD camera 1 picturizes the subject 1 1 in the robot end point position 
Ps, and supplies this picture to the sensor controller 2. The sensor controller 2 extracts 
image characteristic quantity based on the picture from CCD camera 1, and shifts to step 
ST6. Here, CCD camera 1 acquires the picture of the subject 1 1 as shown, for example in 

drawing 6 . 

[0033]In step ST6, six attachment holes processed into the upper surface 1 18 of the 
subject 1 1 are extracted as the image characteristic quantity Fi (i= 1-6), the center 
position of each image characteristic quantity Fi carries out coordinate value (Xi, Yi) 
memory, and the sensor controller 2 ends processing of teaching work. 
[0034]What is necessary is just to use publicly known techniques, such as binarization 
image processing and edge processing, as an instrumentation method of the center 
position of the attachment hole in a picture. Although it attached here and the center 
position of the hole was made into the image characteristic quantity Fi, the area center of 
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gravity of the isolated field of a line segment corner point, a corner, and a binarization 
picture, etc. may use other image characteristic quantity. An operator may give a 
definition artificially by the input means in which neither a mouse nor a keyboard 
illustrates the image characteristic quantity Fi at the time of instruction. 
[0035](Grip operations) A robot device grasps the subject 1 1 under conveyance using the 
coordinates of the image characteristic quantity Fi memorized on the occasion of the 
work, and the robot end point positions Pd and Ps, after teaching work is completed. In 
these grip operations, processings from step ST1 1 to step ST 18 shown in drawing 7 are 
performed. 

[0036]In step ST 1 1, as shown in drawing 8 , CCD camera 1 picturizes a transportation 
area [ of the subject 1 1 ] 10, i.e., conveyor, top, and shifts to step ST12. 
[0037]In step ST12, the sensor controller 2 judges whether the subject 1 1 was detected. 
In order to detect the subject 1 1 and to extract characteristic quantity from the picture, it 
is required for the subject 1 1 whole to enter within the limits of the view of CCD camera 
1. Here, the luminosity of a picture is used for detection of the subject 11. For example, 
when the subject 1 1 is upstream of the conveyor 10 and there is into the visual field range 
of CCD camera 1, the luminosity of the picture acquired with CCD camera 1 hardly 
changes. [ no ] On the other hand, if the subject 1 1 is conveyed by conveyor and it enters 
in the visual field range of CCD camera 1 , the luminosity will change. Then, it judges 
with the sensor controller 2 providing the window which is not illustrated in the suitable 
position in a picture, and the subject 1 1 being in a visual field range, when the luminosity 
in the window is beyond a predetermined threshold, and when luminosity is less than a 
predetermined threshold, it judges with there being nothing within the limits of the view. 
And when the sensor controller 2 detects the subject 1 1, it shifts to step ST 13, and when 
it does not detect, it returns to step ST1 1 . 

[0038]image-characteristic-quantity fi(t) (i=l in the time T from a picture when CCD 
camera 1 picturized the sensor controller 2 in step ST 13, and 2 ...) — it extracts and shifts 
to step ST14. Here, drawing 9 is a figure showing the picture picturized at the time of 
execution of grip operations. It is necessary to extract the image characteristic quantity fi 
(t) from the picture picturized at the time t at the time of grip operations. However, there 
is a thing which should extract essentially by a certain cause and for which the image 
characteristic quantity f3 (t) is extracted for example, it does not come out. Then, the 
characteristic quantity fl (t) - f7 (t) need to perform corresponding point searching which 
asks for any of Fl - F6 which show drawing 6 are supported. 

[0039]This corresponding point searching can be performed with what is called checking 
methods, such as dynamic programming, a largest faction method, etc. which various 
methods are proposed, for example, are indicated to "robot vision" (Masahiko Yanaida 
work, Shokodo, 1990). If Fl - the thing corresponding to F6 are obtained out of fl-f7, the 
following image characteristic quantity can once be calculated easily. 
[0040]What is necessary is just to specifically extract the image characteristic quantity fi 
(t+deltat) in the next time (t+deltat) of the time t near the image characteristic quantity fi 
in the inside of a previous picture (t), if cycle time of the repetition processings from step 
ST 13 to step ST 16 is set to deltat. Thereby, the stability of characteristic quantity 
extraction can ease increase and the burden of computation. 

[0041]In step ST 14, the sensor controller 2, It judges whether the error vector E (t) in the 
time t is small enough, or the judging standard of ||E(t) ||<epsilon is specifically met, 
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when small enough, it shifts to step ST 17, and when not small enough, it shifts to step 
ST15. 

[0042]Here, the error vector E (t) is searched for based on the image characteristic 
quantity fi (t) extracted from the picture of present CCD camera 1 , and the image 
characteristic quantity Fi memorized by step ST6 mentioned above. 
[0043]In this invention, literature B.Espiau et al., A New Approach to Visual Servoing in 
Robotics, IEEE Trans. Robotics and Automation, 8 (3), and 313-326-1992, Literature 
F.Bensalha et al. and Real Time Visual Tracking using the Generalized Likelihood Ratio 
Test,Int.Conf.Automation,Robotics and. Based on Computer Vision and the technique 
indicated to 1379-1383-1994, the error vector E (t) and the speed commanding value 
vector V (t) mentioned later are computed. 

[0044]It asks for characteristic quantity deviation vector e (t) by the following formulas 
(1) using image-characteristic-quantity fi (t) = (Xi (t), Yi (t)) obtained from the picture 
picturized at image-characteristic-quantity Fi= at the time of instruction (Xi, Yi), and the 
time t. 
[0045] 
[Equation 1] 

" t(t)-x, 
y,(t)-y, 
x 2 (t)-x 2 

e(t)= y 2 (t)-y s ■ - • (1) 



x*(t)-x N 
^y N (t)-y* J 

[0046](xi, yi) and (xi(t), and yi (t)) are expressed with a pixel value (Xi, Yi), and change 
(Xi(t) and Yi (t)) into the value in a camera coordinate system, respectively. N expresses 
the number of image characteristic quantity. 

[0047]The error vector E (t) is expressed with a formula (2) using above-mentioned 
characteristic quantity deviation vector e (t). 
[0048] 
[Equation 2] 

E(t)=L T+ e<t) ■ . ■ (2) 

[0049]Here, L T+ is a false inverse matrix of picture Jacobian L T calculated from the 
image characteristic quantity F and F (t). 

[0050]It judges with the sensor controller 2 having followed the subject 1 1 which 
exercises, when the judging standard of ||E(t) ||<epsilon was met. epsilon is scalar 
quantity beforehand given by an operator. Or as long as it expresses E=(E1 E2 E3 E4 E5 
E6) T , the judging standard shown in a formula (3) may be used. 
[0051] 
[Equation 3] 

maxEi<£ - ■ ■ (3) 

[0052]In step ST 15, the robot controller 5 computes the speed commanding value vector 
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V (t), and shifts to step ST 16. The speed commanding value vector V contains the 
advancing-side-by-side speed (t x , t y , t z ) and revolving speed (omega x , omega y , omega z ) of 
the robot end point position P, It is expressed V=(t x t y t z omega x omega y omega z ) T . And the 
speed commanding value vector V (t+deltat) in time (t+deltat) is searched for from a 
formula (4). 
[0053] 
[Equation 4] 

V(t+At)=-AL T+ e(t)-T°(t) ■ ■ ■ (4) 

[0054]lambda is a speed gain. The rightmost paragraph of the right-hand side of a 
formula (4) is a point estimate of the motion velocity of the subject 11, and is searched 
for from a formula (5). 
[0055] 
[Equation 5] 

ret)- Ett)-Eft-At) _ V(t) . . , m 

[0056]The robot controller 5 computes error vector E (t) and the speed commanding 
value vector V (t) as mentioned above. 

[0057]In step ST 16, the robot controller 5 generates trajectory generation, and joint 
angular velocity and angular acceleration based on the computed speed commanding 
value vector V (t), controls the robot arm 4 according to these, and returns to step ST13. 
[0058]In step ST 17 when E (t) judges with it being small enough by step ST 14, the robot 
controller 5 judges with the gripper 3 having followed the subject 1 1 , and starts approach 
to the subject 1 1 . In the following explanation, it is called visual servo by processing the 
flow chart shown in drawing 7 to make the gripper 3 follow a subject. 
[0059]Here, if time which shifted to step ST14 is made into time ti, as shown, for 
example in drawing.] 0, as for a speed commanding value vector, a position vector of 
Vactuai (ti) and a robot end point position will be set to P (ti). 

[0060]In time ti, since the robot end point position P follows the subject 1 1, its speed 
commanding value vector V actU ai at that time (ti) is equal to a velocity vector of the 
subject 1 1 . Therefore, the robot controller 5 generates a speed pattern on which a speed 
pattern for moving by Po from the end point position P (ti) was made to superimpose 
speed commanding value vector V actU ai (ti) mentioned above. Po is the positional attitude 
transformation matrix shown by teaching work. A speed pattern for moving by Po is 
computable with the conventional techniques, such as a trapezoid speed pattern, for 
example. 

[0061]The robot controller 5 computes P a according to the following formulas noting that 

it takes time T 0 to carry out P 0 part relative displacement. 

[0062] 

P a =P(ti)+P 0 + V ac tuai (ti), T 0 , and the robot controller 5 control the robot arm 4 based on 
computed P a . Thereby, as shown in drawing 1 1 , a robot hand (gripper 3) can be moved to 
position P a which should grasp the subject 1 1 from the end point position P (ti). After 
such approach is completed, it shifts to step ST 18. 

[0063 ]In step ST 18, the robot controller 5 grasps the subject 1 1 by the gripper 3, and ends 
processing. 

[0064]As mentioned above, the robot device can memorize the robot end point positions 
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Ps and Pd at the time of teaching work, and can follow the subject 1 1 under conveyance 
by a visual servo, and can perform grip operations of the subject 1 1 using the robot end 
point positions Ps and Pd. Thereby, the robot device can grasp the subject 1 1 under 
conveyance easily, without [ without it is restrained by attitude position of the subject 1 1 
under conveyance, and ] stopping the conveyor 10. 

[0065]Since work which searches for beforehand physical relationship the installed 
position, CCD camera 1, the gripper 3, and between subject 1 1 is unnecessary, the above- 
mentioned robot device, For example, even if it is a case where installation requirements 
of CCD camera 1 change, the subject 1 1 under conveyance can be grasped only by 
simple teaching work which was mentioned above. Even if it is a case where the subjects 
1 1 which should be grasped differ when it is going to realize the same work by a different 
robot device, CCD camera 1, and the gripper 3, only simple teaching work is similarly. 
[0066]Thereby, the operator can reduce load in the conventional teaching work and 
programming substantially. Even if change arises in CCD camera 1, the gripper 3, and 
the subject 1 1, it can respond flexibly. 

[0067]In an embodiment mentioned above, although it mentioned as an example grasping 
the subject 1 1 currently conveyed by conveyor 10 and it was explained, a robot device 
can be made the same, also when grasping the stationary subject 1 1 .Teaching work is the 
same as explanation mentioned above. 

[0068]In grip operations, in step ST 14 shown in drawing 7 , when the error vector E (t) 
becomes small enough, the speed commanding value vector V (t) becomes zero, and 
speed V ac tuai (ti) of a actual robot end point position becomes zero similarly. 
[0069]Therefore, in step ST 17, the robot controller 5 can grasp the subject 1 1 only using 
change part Po of the present robot end point position P (ti) and a position at the time of 
instruction. That is, position P a which should be grasped is calculated by the following 
formulas. 

[0070]P a =P(ti)+Po and the robot controller 5 can control the robot arm 4 based on 
computed P a , and can grasp the subject 1 1 stationary by the gripper 3. 
[0071](A 2nd embodiment) Below, a 2nd embodiment of this invention is described. The 
same numerals are given to the same part as a 1st embodiment, and the statement is 
omitted about overlapping explanation. 

[0072] A 1st embodiment explained a full flattery state where a subject under conveyance 
and relative physical relationship with the robot end point position P became almost the 
same as those relative physical relationship at the time of teaching work. On the other 
hand, by a 2nd embodiment, when it is not in a full flattery state, it explains that a robot 
device grasps a subject. 

[0073]Here, with a flattery state, it is defined as the state where relative position relation 
was kept constant. For example, in a case where a subject is exercising by speed V 0 , the 
robot end point position P is in a state where it is exercising by speed Vo. Next, two, a 
"full flattery state" and an "imperfect flattery state", are defined as a flattery state. 
[0074JA state which is carrying out fixed time continuation while relative physical 
relationship of the robot end point position P and a subject has been constant value Po, 
i.e., the state of being the same as those relative physical relationship at the time of 
teaching work, is called "full flattery state." A state which is changing fixed time 
continuation into an "imperfect flattery state" while relative physical relationship of the 
robot end point position P and a subject has been constant value P' 0 (!=Po), That is, 
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although there is no change of a position relative between the robot end point position P 
and a subject, the state of differing from relative physical relationship at the time of 
teaching work is said. About these two states, a state of speed of a picture error, the robot 
end point position P, and the robot end point position P and acceleration of the robot end 
point position P is shown in Table 1 . 
[0075] 
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[0076]"In the state of full flattery", relative physical relationship of the robot end point 
position P and a subject is P 0 , and, moreover, the robot end point position P is moved at 
the same speed as a subject. For this reason, the approaching operation can perform grip 
operations by moving approach move Po, maintaining the following movement V. 
[0077]On the other hand, the robot end point position P is moved at the same speed as a 
subject also "in the state of imperfect flattery." However, relative physical relationship P' 0 
of the robot end point position P and a subject differs from physical relationship Po at the 
time of teaching work. For example, as shown in drawing 12 (A), the initial picture of the 
subject 1 1 at the time of a flattery start is not in agreement with a target image. In order to 
make it be in a flattery state early at this time, the speed of the robot end point position Ps 
is large as shown in drawing 13 . As shown in drawing 1 2 (B), the predetermined gap will 
be maintained the picture of the subject 1 1 of a flattery state to the target image. The 
speed of the robot end point position Ps at this time is V ac tuai (ti) as shown in drawing 13 . 
[0078]Then, if relative-position-relation P' 0 can be taught, as shown in drawing 12_ (C), 
the completely same effect as a "full flattery state" can be acquired. Physical relationship 
P'o in an "imperfect flattery state" has **** which changes by change of the motion 
velocity of a subject, change of the gain of a visual servo, etc., and re-instruction is 
required for it at every time. According to this embodiment, re-teaching work is 
performed so that it can respond to an "imperfect flattery state." 

[0079](Re -teaching work and grip operations) In re-teaching work, first, a visual servo is 
used and flattery of a subject is performed. And the robot controller 5 extracts image- 
characteristic-quantity F'i of a subject at this time, and memorizes the coordinates of each 
image-characteristic-quantity F'i. The extracting processing of image-characteristic- 
quantity F'i is the same as processing of step ST6 shown in drawing 3 . 
[0080]Next processings from step ST1 to step ST3 shown in drawing 3 are performed, 
and the gripper 3 is moved to arbitrary positions (robot end point position Ps). Further, 
the robot controller 5 performs a visual servo using image-characteristic-quantity Fi', and 
makes the position of the gripper 3 which reached the flattery state robot end point 
position P's. And the robot controller 5 computes positional attitude trans formation- 
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matrix P'o (=P's _1 Pd) in a flattery state, and memorizes this positional attitude 
transformation-matrix P'o. 

[0081]In grip operations, processings from step ST1 1 to step ST 18 shown in drawing 7 
are performed like a 1st embodiment. What is necessary is to replace with the image 
characteristic quantity Fi and positional attitude conversion Po which were calculated by 
a 1st embodiment, and just to use image-characteristic-quantity F'i and positional attitude 
conversion business sequence P'o at this time. 

[0082]As mentioned above, according to the robot device concerning a 2nd embodiment, 
even if the robot end point position P does not follow the subject 1 1 thoroughly, the 
subject 1 1 can be grasped only by performing easy teaching work mentioned above 
without carrying out complicated adjustment. 

[0083](A 3rd embodiment) According to 1st and 2nd embodiments, although the case of 
stillness movement and linear uniform motion was shown as an example, if this invention 
is used, it is expandable to a linear motion of acceleration or uniform circular motion. 
And processings from step ST1 1 to step ST 17 shown in drawing 7 are performed like [ 
these grip operations ] a 1st embodiment. 

[0084]In the case of a linear motion of acceleration, specifically the robot controller 5, In 
time ti to which the error vector E (t) became small enough, Judge with the robot end 
point position P having been relatively controlled by the position of Po or P'o to the 
subject 11, and V ac t U ai (ti) in this time ti, Velocity component V es timate (t) generated from 
hand acceleration acc ac tuai (ti), The speed commanding value which piled up the speed 
commanding value generated from Po or P'o is generated, the subject 1 1 is approached 
from the robot end point position P in time ti (ti), and the subject 1 1 is grasped by the 
gripper 3. 

[0085]That is, the robot controller 5 is controlled to move the gripper 3 by Po from the 
robot end point position Ps when it changes into a flattery state to be shown in drawing 
14 . The robot controller 5 computes P a according to the following formulas noting that it 
takes time T' 0 to carry out P 0 part relative displacement at this time. 
[0086] 
[Equation 6] 



[0087]Although the case where a subject was grasped was mentioned as the example and 
the embodiment mentioned above explained it, this invention is not limited to this. For 
example, even if it is a case where attachment of parts, etc. are carried out to the subject 
1 1 , it is applicable similarly. 



[Effect of the InventionJAccording to a robot device concerning this invention, and a 
method for controlling the same, the operator should just perform the following two 
things as teaching work. A gripping mechanism is moved to the 1st end point position, 
and a subject is made 1st to grasp here. From a gripping mechanism, a subject is made to 
release and a gripping mechanism is moved [ 2nd ] to the 2nd end point position into an 
imaging range. The coordinates of the characteristic quantity of the subject in the 1st and 
2nd end point positions and the 2nd end point position which are memorized at this time 
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0 



[0088] 



are used in order to follow a subject and to grasp this. That is, an operator only performs 
easy teaching work and can make the subject under movement to a robot device grasp. 



TECHNICAL FIELD 



[Field of the Invention] This invention relates to a robot device and a method for 
controlling the same, and relates to a robot device which grasps the object under 
movement especially, and a method for controlling the same. 



PRIOR ART 

[Description of the Prior ArtjWhen working to a subject using a robot (for example, 
objective grasping), information, including the position of a subject, a posture, etc., is 
acquired by a sensor, and many examples which work based on the sensor information 
are put in practical use. As such an equipment configuration, it divides roughly and, 
generally there are the following two methods. 

[0003]For example, there are some which detect the position of a subject and a posture 
from the picture acquired by picturizing with the camera which had the subject which has 
a conveyor top conveyed fixed, and transmit the information to a robot as indicated to 
JP,8-63214,A. This robot is working grasping etc. from the position information on a 
subject. 

[0004]Sensors, such as a camera, are attached to the hand of a robot, the target positional 
attitude is detected from the picture from a camera as mentioned above, and there are 
some which follow the subject which moves based on the information, and work grasping 
etc. 



EFFECT OF THE INVENTION 



[Effect of the InventionJAccording to a robot device concerning this invention, and a 
method for controlling the same, the operator should just perform the following two 
things as teaching work. A gripping mechanism is moved to the 1st end point position, 
and a subject is made 1st to grasp here. From a gripping mechanism, a subject is made to 
release and a gripping mechanism is moved [ 2nd ] to the 2nd end point position into an 
imaging range. The coordinates of the characteristic quantity of the subject in the 1st and 
2nd end point positions and the 2nd end point position which are memorized at this time 
are used in order to follow a subject and to grasp this. That is, an operator only performs 
easy teaching work and can make the subject under movement to a robot device grasp. 



TECHNICAL PROBLEM 



[Problem(s) to be Solved by the InventionJHowever, when realizing the work of grasping 
of a subject, etc. using the camera installed apart from the robot like the former, the work 
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which asks for the installed position of a robot and a camera correctly beforehand is 
needed. It is necessary to perform the camera calibration which computes a focal 
distance, a lens strain coefficient, etc. of the camera beforehand. 
[0006]In addition, in the example in JP,8-63214,A etc., it is necessary to install so that 
the optic axis of a camera may generally become vertical still more nearly parallel to the 
transportation direction of a conveyor at a conveyor flat surface in the x axis or the y-axis 
in a screen. Since the subject is running by the band conveyor etc., in addition to the 
position attitude information of the subject detected by the picture from a camera, another 
means (for example, encoder of a conveyor) needs to detect the speed information of a 
conveyor, etc., and it is necessary to generate the command value to a robot based on 
both information. The orbit at the time of a robot grasping a subject from the installation 
position information of a camera and a robot, the position information on a subject, and 
the speed information of a conveyor must be generated. Therefore, there are problems, 
like the procedure for teaching work and trajectory generation becomes very complicated. 
[0007]It can say such a problem that the same may be said of the example of the latter 
which installed the camera to the hand of the robot in conventional technology. That is, 
when a subject is grasped by the gripper etc. which were attached to the hand of a robot 
based on the position attitude information of the subject detected from the picture from a 
camera, What is called a hand eye calibration that needs to change the position attitude 
information of the subject acquired from the camera into the position attitude information 
seen from the standard coordinates of the robot, and searches for the physical relationship 
of a robot hand and a camera precisely beforehand is needed. When sensors, such as a 
camera, are especially installed in a robot hand, a camera position relation with a robot 
hand frequently changes with the malfunctions under work, etc., and it is necessary to do 
calibration work again in that case. When the subject is moving like the former example, 
since the position attitude information of the subject seen from the robot normal 
coordinate changes every moment, it becomes very complicated [ the trajectory 
generation for grasping ]. 

[0008]In the former, even if the method of the former and the latter changes neither a 
component (the installation condition of a conveyor, a bearer rate, or the installed 
position relation between a camera and a robot), nor the shape of a subject which should 
be grasped, Whenever the robot and sensor to be used changed, the above-mentioned 
highly precise calibration and complicated teaching work needed to be performed. 
[0009]This invention is proposed in order to cancel the problem mentioned above, and it 
is a thing. 

The purpose is to provide a robot device which can save the time and effort which 
performs work and a calibration, and can make predetermined work do on a subject 
easily, and a method for controlling the same. 



MEANS 



[Means for Solving the ProblemJA holding means from which the invention according to 
claim 1 has a gripping mechanism which grasps a subject and which said gripping 
mechanism comprised movable, An imaging means which is fixed movable with a 
gripping mechanism of said holding means, and picturizes a subject, A feature amount 
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extracting means which extracts coordinates of one or more characteristic quantity from a 
picture of a subject which said imaging means picturized, A position of a gripping 
mechanism at a time of making said gripping mechanism grasp said subject is memorized 
as the 1st end point position, A position of said gripping mechanism when moving said 
gripping mechanism within limits which can make said subject able to open wide and in 
which said imaging means can picturize a subject is memorized as the 2nd end point 
position, A memory measure which memorizes coordinates of each characteristic 
quantity of said subject which said imaging means picturized in said 2nd end point 
position, and said feature amount extracting means extracted, A calculating means which 
computes a move procession for moving from said 2nd end point position to said end 
point position of 1, Coordinates of each characteristic quantity of a subject under 
movement extracted by said feature amount extracting means by moving said holding 
means so that it may be in agreement with coordinates of each characteristic quantity 
memorized by said memory measure. When a follow-up control means which carries out 
control which said gripping mechanism is made to follow to a subject under said 
movement, and a subject while said gripping mechanism is exercising are followed, 
Based on a move procession which said calculating means computed, said gripping 
mechanism is brought close to a subject under said movement, and it has a grasping 
control means which controls said holding means to make said subject grasp. 
[001 l]The robot device according to claim 1 grasps a subject which exercises relatively 
to the robot device concerned. Therefore, a case where said robot device exercises and 
said subject is being fixed, when said robot device and said subject are exercising at a 
speed different, respectively, and not only when said robot device is fixed and said 
subject exercises, but when, it can apply. Although said subject may be carrying out the 
acceleration straight-line motion of linear uniform motion, the uniform circular motion, 
etc. relatively to said robot device, it may be standing it still relatively. 
[0012]First, the operator needs to perform the following two things as teaching work. A 
gripping mechanism is moved to the 1st end point position, and a subject is made 1st to 
grasp here. A subject is made to release from a gripping mechanism and an imaging 
means moves [ 2nd ] a gripping mechanism within limits which can picturize a subject to 
the 2nd end point position. Such teaching work is performed and a memory measure is 
made to memorize the 1st and 2nd end point positions. A memory measure is made to 
also memorize coordinates of one or more characteristic quantity extracted from a picture 
which an imaging means picturized in the 2nd end point position. An imaging means 
serves as a sensor of a gripping mechanism, and it is being fixed so that it may move with 
a holding means. A move procession for moving from the 2nd end point position to the 
1st end point position is computed. 

[0013]If such teaching work is completed, grip operations which grasp a subject under 
movement can be performed. A gripping mechanism is moved so that coordinates of each 
characteristic quantity of a subject under movement may be in agreement with 
coordinates of each characteristic quantity memorized by said memory measure. That is, 
what is called a visual servo is performed so that a subject while a gripping mechanism is 
exercising may be followed. Relative physical relationship of a subject at this time and a 
gripping mechanism is the same as a subject at the time of teaching work, and that 
relative of a gripping mechanism. When a subject while said gripping mechanism is 
exercising is followed, based on the 1st move procession that said calculating means 
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computed, said gripping mechanism is brought close to a subject under said movement, 
and said subject is made to grasp. 

[0014]It is desirable for a gripping mechanism to follow a subject under movement 
thoroughly in a case where a visual servo is being performed. However, a gripping 
mechanism may follow, separating prescribed distance to a subject under movement. In 
such a case, it is preferred that re-teaching work which is indicated to claim 2 is 
performed. While said gripping mechanism separates said memory measure prescribed 
distance to a subject under said movement by said follow-up control means, when it 
follows, specifically, Coordinates of each characteristic quantity of said subject which 
said feature amount extracting means extracted are newly memorized, What is necessary 
is just to memorize a position of said gripping mechanism when said follow-up control 
means makes said gripping mechanism follow a subject thoroughly from said 2nd end 
point position using coordinates of each newly memorized characteristic quantity as the 
2nd new end point position. 

[0015]The invention according to claim 3 is the control method of a robot device which 
grasps a subject by a gripping mechanism constituted movable, A position of a gripping 
mechanism at a time of making said gripping mechanism grasp said subject is memorized 
as the 1st end point position, A position of said gripping mechanism when making said 
subject open wide and moving said gripping mechanism within limits which can picturize 
a subject is memorized as the 2nd end point position, In said 2nd end point position, 
picturize said subject, and coordinates of one or more characteristic quantity are extracted 
from a picture of said picturized subject, In the state where memorized coordinates of 
each of said extracted characteristic quantity, computed a move procession for moving 
from said 2nd end point position to said end point position of 1 , and it was fixed movable 
with said gripping mechanism. Picturize a subject under movement and coordinates of 
one or more characteristic quantity are extracted from a picture of a subject under said 
picturized movement, Coordinates of each characteristic quantity of a subject under said 
extracted movement by moving said gripping mechanism so that it may be in agreement 
with coordinates of each of said characteristic quantity memorized. When said gripping 
mechanism is made to follow a subject under said movement and a subject while said 
gripping mechanism is exercising is followed, based on said computed move procession, 
said gripping mechanism is brought close to a subject under said movement, Said 
gripping mechanism is made to grasp said subject. 

[0016]This invention is a method for controlling a robot device indicated to claim 1, and 
the operator should just perform two things mentioned above as teaching work. That is, 
only by an operator performing two easy teaching work, a gripping mechanism can be 
made to be able to follow a subject which is exercising and the subject can be grasped. 
[0017]It is desirable for a gripping mechanism to follow a subject under movement 
thoroughly in a case where a visual servo is being performed. However, a gripping 
mechanism may follow, separating prescribed distance to a subject under movement. In 
such a case, it is preferred that re-teaching work which is indicated to claim 4 is 
performed. While said gripping mechanism specifically separates prescribed distance to a 
subject under said movement, when it makes said gripping mechanism follow a subject 
under said movement, and it follows, it is in a state fixed movable with said gripping 
mechanism, Picturize a subject under movement and coordinates of one or more 
characteristic quantity are extracted from a picture of a subject under said picturized 
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movement, Newly memorize coordinates of each characteristic quantity of said said 
extracted subject, and coordinates of each of said newly memorized characteristic 
quantity are used, What is necessary is just to memorize a position of said gripping 
mechanism as the 2nd new end point position, when said gripping mechanism is moved 
so that said gripping mechanism may follow a subject thoroughly from said 2nd end point 
position, and followed thoroughly. 
[0018] 

[Embodiment of the Invention] Hereafter, it explains in detail, referring to drawings for an 
embodiment of the invention. 

[0019](A 1st embodiment) As shown in drawirig J ., the robot device concerning this 
embodiment grasps the subject 1 1 currently conveyed by conveyor 10 in the state where 
it was fixed. This invention is not limited to such an embodiment, and if the subject 1 1 is 
exercising relatively to a robot device, it is applicable. That is, they may be a case where 
a robot device exercises and the subject 1 1 is standing it still, and a case where it is 
exercising at the speed from which a robot device and the subject 1 1 differ. 2nd and 3rd 
embodiments are also the same. 
[0020]A robot device is provided with the following. 
CCD camera 1 which picturizes the subject 11. 

The sensor controller 2 which extracts characteristic quantity from the picture which 

CCD camera 1 picturized. 

The gripper 3 which grasps the subject 1 1 . 

The robot controller 5 which performs the robot arm 4 which moves the gripper 3 in the 
direction of a three dimension, coordinate conversion, trajectory generation, etc., has a 
servo circuit, amplifier, etc. which are not illustrated, and controls operation of the 
gripper 3 or the robot arm 4. 

[0021]CCD camera 1 has adhered near the gripper 3 which is a tip of the robot arm 4. 
The robot controller 5 computes error vector E (t) and the speed commanding value 
vector V (t) based on the image characteristic quantity Fi obtained with the sensor 
controller 2, and controls movement of the robot arm 4, and grasping of the subject 1 1 by 
the gripper 3 according to these values. The conveyor 10 conveys the subject 1 1 at a 
fixed speed. 

[0022]Before the robot device of such composition actually grasps the subject 1 1 
currently conveyed, teaching work for grasping the subject 1 1 is performed. 
[0023](Teaching work) First, an operator makes a robot device grasp the subject 11, and 
teaches the robot end point position P. The position which should be grasped may be 
restricted by the shape of gripping mechanisms, such as the gripper 3, and the subject 11. 
The position which should be grasped for next processing (for example, attach other parts 
to the subject 1 1, or supply to a parts box) may be specified beforehand. 
[0024]Here, as shown, for example in drawing 2 , the subject 1 1 was formed in 7 
prismatic forms, and is provided with the seven sides 111-117. Six attachment holes (for 
example, tapped hole) for attaching the parts which are not illustrated are formed in the 
upper surface 1 1 8 of the subject 1 1 . Such an attachment hole is not formed in the 
undersurface 1 19 of the subject 1 1 . As for the portion by which the subject 1 1 is grasped, 
it is preferred that they are the side 1 1 1,1 15 in which it faces among seven sides in 
consideration of stability when grasped, or the side 113,116. Although the subject 1 1 of 
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the above-mentioned shape was used in this embodiment, this invention may not be 
limited to this shape and the subjects 1 1 may be other shape. And teaching work is 
specifically performed according to processing of step ST1 to step ST6 shown in drawing 
3. 

[0025]In step ST1, an operator shifts to step ST2, as the gripper 3 grasps the side 111,115 
or the side 113,116 of the subject 1 1 using the teaching pendant etc. which are not 
illustrated. 

[0026]In step ST2, as shown in drawing -4, the gripper 3 is in the state which grasped the 
subject 11, and the robot controller 5 memorizes, the position, i.e., robot end point 
position Pd, of the gripper 3 in robot standard coordinates, and shifts to step ST3. The 
"position" said here is expressed with the parameter of 6 flexibility including a posture. 
The parameter of 6 flexibility including a posture is called "position" like the following. 
[0027]In step ST3, the robot controller 5 makes the subject 1 1 which the gripper 3 
grasped release, and moves the gripper 3 to arbitrary positions, and shifts to step ST4. 
[0028]It requires that the position which the gripper 3 moves is in the visual field range 
of CCD camera 1. That is, it is a position where the characteristic quantity of the subject 
1 1 is extracted from the picture picturized with CCD camera 1 . It is a position where 
characteristic quantity which should be extracted, such as a position to which the subject 
1 1 is not settled in the view of CCD camera 1 , and a position in which CCD camera 1 
picturizes the subject 1 1 from just beside, is specifically suitably obtained from a picture. 
[0029]In step ST4, the robot controller 5 memorizes, as shown in drawing 5 . the position 
Ps, i.e., the robot end point position, of the gripper 3 in the present robot standard 
coordinates, and it shifts to step ST5. 

[0030]Here, the following formulas will be realized if the positional attitude 
transformation matrix for changing into Pd from the robot end point position Ps is made 
into Po- 

[0031]Ps-Po = PdthereforePo = Ps _1 and the Pd robot controller 5 compute positional 
attitude transformation-matrix Po at this time, and memorizes this. The robot controller 5 
may compute positional attitude transformation-matrix Po in the case of the grip 
operations mentioned later. 

[0032]In step ST5, CCD camera 1 picturizes the subject 1 1 in the robot end point position 
Ps, and supplies this picture to the sensor controller 2. The sensor controller 2 extracts 
image characteristic quantity based on the picture from CCD camera 1 , and shifts to step 
ST6. Here, CCD camera 1 acquires the picture of the subject 1 1 as shown, for example in 

drawing 6 . 

[0033]In step ST6, six attachment holes processed into the upper surface 1 18 of the 
subject 1 1 are extracted as the image characteristic quantity Fi (i= 1-6), the center 
position of each image characteristic quantity Fi carries out coordinate value (Xi, Yi) 
memory, and the sensor controller 2 ends processing of teaching work. 
[0034]What is necessary is just to use publicly known techniques, such as binarization 
image processing and edge processing, as an instrumentation method of the center 
position of the attachment hole in a picture. Although it attached here and the center 
position of the hole was made into the image characteristic quantity Fi, the area center of 
gravity of the isolated field of a line segment corner point, a corner, and a binarization 
picture, etc. may use other image characteristic quantity. An operator may give a 
definition artificially by the input means in which neither a mouse nor a keyboard 
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illustrates the image characteristic quantity Fi at the time of instruction. 

[0035](Grip operations) A robot device grasps the subject 1 1 under conveyance using the 

coordinates of the image characteristic quantity Fi memorized on the occasion of the 

work, and the robot end point positions Pd and Ps, after teaching work is completed. In 

these grip operations, processings from step ST11 to step ST18 shown in drawing 7 are 

performed. 

[0036]In step ST 1 1, as shown in drawing 8 , CCD camera 1 picturizes a transportation 
area [ of the subject 1 1 ] 10, i.e., conveyor, top, and shifts to step ST12. 
[0037]In step ST12, the sensor controller 2 judges whether the subject 1 1 was detected. 
In order to detect the subject 1 1 and to extract characteristic quantity from the picture, it 
is required for the subject 1 1 whole to enter within the limits of the view of CCD camera 
1. Here, the luminosity of a picture is used for detection of the subject 11. For example, 
when the subject 1 1 is upstream of the conveyor 10 and there is into the visual field range 
of CCD camera 1, the luminosity of the picture acquired with CCD camera 1 hardly 
changes. [ no ] On the other hand, if the subject 1 1 is conveyed by conveyor and it enters 
in the visual field range of CCD camera 1, the luminosity will change. Then, it judges 
with the sensor controller 2 providing the window which is not illustrated in the suitable 
position in a picture, and the subject 1 1 being in a visual field range, when the luminosity 
in the window is beyond a predetermined threshold, and when luminosity is less than a 
predetermined threshold, it judges with there being nothing within the limits of the view. 
And when the sensor controller 2 detects the subject 1 1, it shifts to step ST 13, and when 
it does not detect, it returns to step ST1 1 . 

[0038]imagc-charactcristic-quantity fi(t) (i=l in the time T from a picture when CCD 
camera 1 picturized the sensor controller 2 in step ST13, and 2 ...) ~ it extracts and shifts 
to step ST14. Here, drawing 9 is a figure showing the picture picturized at the time of 
execution of grip operations. It is necessary to extract the image characteristic quantity fi 
(t) from the picture picturized at the time t at the time of grip operations. However, there 
is a thing which should extract essentially by a certain cause and for which the image 
characteristic quantity f3 (t) is extracted for example, it does not come out. Then, the 
characteristic quantity fl (t) - f7 (t) need to perform corresponding point searching which 
asks for any of Fl - F6 which show drawing 6 are supported. 

[0039]This corresponding point searching can be performed with what is called checking 
methods, such as dynamic programming, a largest faction method, etc. which various 
methods are proposed, for example, are indicated to "robot vision" (Masahiko Yanaida 
work, Shokodo, 1990). If Fl - the thing corresponding to F6 are obtained out of fl-f7, the 
following image characteristic quantity can once be calculated easily. 
[0040]What is necessary is just to specifically extract the image characteristic quantity fi 
(t+deltat) in the next time (t+deltat) of the time t near the image characteristic quantity fi 
in the inside of a previous picture (t), if cycle time of the repetition processings from step 
ST 13 to step ST 16 is set to deltat. Thereby, the stability of characteristic quantity 
extraction can ease increase and the burden of computation. 

[0041]In step ST 14, the sensor controller 2, It judges whether the error vector E (t) in the 
time t is small enough, or the judging standard of ]|E(t) ||<epsilon is specifically met, 
when small enough, it shifts to step ST 17, and when not small enough, it shifts to step 
ST15. 

[0042]Here, the error vector E (t) is searched for based on the image characteristic 
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quantity fi (t) extracted from the picture of present CCD camera 1 , and the image 
characteristic quantity Fi memorized by step ST6 mentioned above. 
[0043]In this invention, literature B.Espiau et al., A New Approach to Visual Servoing in 
Robotics, IEEE Trans. Robotics and Automation, 8 (3), and 313-326-1992, Literature 
F.Bensalha et al. and Real Time Visual Tracking using the Generalized Likelihood Ratio 
Test,Int.Conf.Automation,Robotics and. Based on Computer Vision and the technique 
indicated to 1379-1383-1994, the error vector E (t) and the speed commanding value 
vector V (t) mentioned later are computed. 

[0044]It asks for characteristic quantity deviation vector e (t) by the following formulas 
(1) using image-characteristic-quantity fi (t) = (Xi (t), Yi (t)) obtained from the picture 
picturized at image-characteristic-quantity Fi= at the time of instruction (Xi, Yi), and the 
time t. 
[0045] 
[Equation 1] 

^x,(t)-x, " 
V,(t)-y, 
x 2 (t)-x 2 

e(t)= y 2 (t)-y s ■ - ■ (1) 



x*(t)-x N 
y N (t)-y* 

[0046](xi, yi) and (xi(t), and yi (t)) are expressed with a pixel value (Xi, Yi), and change 
(Xi(t) and Yi (t)) into the value in a camera coordinate system, respectively. N expresses 
the number of image characteristic quantity. 

[0047]The error vector E (t) is expressed with a formula (2) using above-mentioned 
characteristic quantity deviation vector e (t). 
[0048] 
[Equation 2] 

E(t)=L T+ e(t) ■ • ■ (2) 

[0049]Here, L T+ is a false inverse matrix of picture Jacobian L T calculated from the 
image characteristic quantity F and F (t). 

[0050]It judges with the sensor controller 2 having followed the subject 1 1 which 
exercises, when the judging standard of ||E(t) ||<epsilon was met. epsilon is scalar 
quantity beforehand given by an operator. Or as long as it expresses E=(E1 E2 E3 E4 E5 
E6) T , the judging standard shown in a formula (3) may be used. 
[0051] 
[Equation 3] 

maxEi<£ ■ ■ ■ (3) 

i 

[0052]In step ST 15, the robot controller 5 computes the speed commanding value vector 
V (t), and shifts to step ST 16. The speed commanding value vector V contains the 
advancing-side-by-side speed (t x , t y , t z ) and revolving speed (omega x , omega y , omega z ) of 
the robot end point position P, It is expressed V=(t x t y t z omega x omega y omega z ) T . And the 
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speed commanding value vector V (t+deltat) in time (t+deltat) is searched for from a 

formula (4). 

[0053] 

[Equation 4] 

V(t+At)=-AL T *e(t)-T°(t) ■ ■ • (4) 

[0054]lambda is a speed gain. The rightmost paragraph of the right-hand side of a 
formula (4) is a point estimate of the motion velocity of the subject 11, and is searched 
for from a formula (5). 
[0055] 



[Equation 5] 

f^ t)= E(t)-E(t-At) 



°(t)= A ' t ' -V(t) ■ ■ ■ (5) 

[0056]The robot controller 5 computes error vector E (t) and the speed commanding 
value vector V (t) as mentioned above. 

[0057]In step ST 16, the robot controller 5 generates trajectory generation, and joint 
angular velocity and angular acceleration based on the computed speed commanding 
value vector V (t), controls the robot arm 4 according to these, and returns to step ST13. 
[0058]In step ST 17 when E (t) judges with it being small enough by step ST 14, the robot 
controller 5 judges with the gripper 3 having followed the subject 11, and starts approach 
to the subject 1 1 . In the following explanation, it is called visual servo by processing the 
flow chart shown in drawing 7 t o make the gripper 3 follow a subject. 
[0059]Here, if time which shifted to step ST14 is made into time ti, as shown, for 
example in drawing 10 , as for a speed commanding value vector, a position vector of 
Vactuai (ti) and a robot end point position will be set to P (ti). 

[0060]In time ti, since the robot end point position P follows the subject 1 1, its speed 
commanding value vector V ac tuai at that time (ti) is equal to a velocity vector of the 
subject 1 1 . Therefore, the robot controller 5 generates a speed pattern on which a speed 
pattern for moving by Po from the end point position P (ti) was made to superimpose 
speed commanding value vector V ac tuai (ti) mentioned above. Po is the positional attitude 
transformation matrix shown by teaching work. A speed pattern for moving by Po is 
computable with the conventional techniques, such as a trapezoid speed pattern, for 
example. 

[0061]The robot controller 5 computes P a according to the following formulas noting that 

it takes time T 0 to carry out P 0 part relative displacement. 

[0062] 

P a =P(ti)+Po+ Vactuai (ti), To, and the robot controller 5 control the robot arm 4 based on 
computed P a . Thereby, as shown in drawing 11 , a robot hand (gripper 3) can be moved to 
position P a which should grasp the subject 1 1 from the end point position P (ti). After 
such approach is completed, it shifts to step ST 18. 

[0063]In step ST18, the robot controller 5 grasps the subject 1 1 by the gripper 3, and ends 
processing. 

[0064]As mentioned above, the robot device can memorize the robot end point positions 
Ps and Pd at the time of teaching work, and can follow the subject 1 1 under conveyance 
by a visual servo, and can perform grip operations of the subject 1 1 using the robot end 
point positions Ps and Pd. Thereby, the robot device can grasp the subject 1 1 under 
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conveyance easily, without [ without it is restrained by attitude position of the subject 1 1 
under conveyance, and ] stopping the conveyor 10. 

[0065]Since work which searches for beforehand physical relationship the installed 
position, CCD camera 1, the gripper 3, and between subject 1 1 is unnecessary, the above- 
mentioned robot device, For example, even if it is a case where installation requirements 
of CCD camera 1 change, the subject 1 1 under conveyance can be grasped only by 
simple teaching work which was mentioned above. Even if it is a case where the subjects 
1 1 which should be grasped differ when it is going to realize the same work by a different 
robot device, CCD camera 1, and the gripper 3, only simple teaching work is similarly. 
[0066]Thereby, the operator can reduce load in the conventional teaching work and 
programming substantially. Even if change arises in CCD camera 1, the gripper 3, and 
the subject 1 1, it can respond flexibly. 

[0067]In an embodiment mentioned above, although it mentioned as an example grasping 
the subject 1 1 currently conveyed by conveyor 10 and it was explained, a robot device 
can be made the same, also when grasping the stationary subject 1 1. Teaching work is the 
same as explanation mentioned above. 

[0068]In grip operations, in step ST 14 shown in drawing 7 , when the error vector E (t) 
becomes small enough, the speed commanding value vector V (t) becomes zero, and 
speed V a ctuai (ti) of a actual robot end point position becomes zero similarly. 
[0069]Therefore, in step ST 17, the robot controller 5 can grasp the subject 1 1 only using 
change part Po of the present robot end point position P (ti) and a position at the time of 
instruction. That is, position P a which should be grasped is calculated by the following 
formulas. 

[0070]P a =P(ti)+Po and the robot controller 5 can control the robot arm 4 based on 
computed P a , and can grasp the subject 1 1 stationary by the gripper 3. 
[0071](A 2nd embodiment) Below, a 2nd embodiment of this invention is described. The 
same numerals are given to the same part as a 1st embodiment, and the statement is 
omitted about overlapping explanation. 

[0072JA 1st embodiment explained a full flattery state where a subject under conveyance 
and relative physical relationship with the robot end point position P became almost the 
same as those relative physical relationship at the time of teaching work. On the other 
hand, by a 2nd embodiment, when it is not in a full flattery state, it explains that a robot 
device grasps a subject. 

[0073]Here, with a flattery state, it is defined as the state where relative position relation 
was kept constant. For example, in a case where a subject is exercising by speed Vo, the 
robot end point position P is in a state where it is exercising by speed V 0 . Next, two, a 
"full flattery state" and an "imperfect flattery state", are defined as a flattery state. 
[0074JA state which is carrying out fixed time continuation while relative physical 
relationship of the robot end point position P and a subject has been constant value Po, 
i.e., the state of being the same as those relative physical relationship at the time of 
teaching work, is called "full flattery state." A state which is changing fixed time 
continuation into an "imperfect flattery state" while relative physical relationship of the 
robot end point position P and a subject has been constant value P'o (!=Po), That is, 
although there is no change of a position relative between the robot end point position P 
and a subject, the state of differing from relative physical relationship at the time of 
teaching work is said. About these two states, a state of speed of a picture error, the robot 
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end point position P, and the robot end point position P and acceleration of the robot end 

point position P is shown in Table 1 . 

[0075] 
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[0076]"In the state of full flattery", relative physical relationship of the robot end point 
position P and a subject is Po, and, moreover, the robot end point position P is moved at 
the same speed as a subject. For this reason, the approaching operation can perform grip 
operations by moving approach move Po, maintaining the following movement V. 
[0077]On the other hand, the robot end point position P is moved at the same speed as a 
subject also "in the state of imperfect flattery." However, relative physical relationship P' 0 
of the robot end point position P and a subject differs from physical relationship P 0 at the 
time of teaching work. For example, as shown in drawing 12 (A), the initial picture of the 
subject 1 1 at the time of a flattery start is not in agreement with a target image. In order to 
make it be in a flattery state early at this time, the speed of the robot end point position Ps 
is large as shown in drawing 13 . As shown in drawing 12 (B), the predetermined gap will 
be maintained the picture of the subject 1 1 of a flattery state to the target image. The 
speed of the robot end point position Ps at this time is V ac tuai (ti) as shown in drawing 13 . 
[0078]Then, if relative-position-relation P' 0 can be taught, as shown in drawing 12 (C), 
the completely same effect as a "full flattery state" can be acquired. Physical relationship 
P'o in an "imperfect flattery state" has **** which changes by change of the motion 
velocity of a subject, change of the gain of a visual servo, etc., and re-instruction is 
required for it at every time. According to this embodiment, re-teaching work is 
performed so that it can respond to an "imperfect flattery state." 

[0079](Re -teaching work and grip operations) In re-teaching work, first, a visual servo is 
used and flattery of a subject is performed. And the robot controller 5 extracts image- 
characteristic-quantity F'i of a subject at this time, and memorizes the coordinates of each 
image-characteristic-quantity F'i. The extracting processing of image-characteristic- 
quantity F'i is the same as processing of step ST6 shown in drawing 3 . 
[0080]Next processings from step ST1 to step ST3 shown in drawing 3 are performed, 
and the gripper 3 is moved to arbitrary positions (robot end point position Ps). Further, 
the robot controller 5 performs a visual servo using image-characteristic-quantity Fi', and 
makes the position of the gripper 3 which reached the flattery state robot end point 
position P's. And the robot controller 5 computes positional attitude transformation- 
matrix P'o (=P's _1 Pd) in a flattery state, and memorizes this positional attitude 
transformation-matrix P'o. 

[0081]In grip operations, processings from step ST1 1 to step ST 18 shown in drawing 7 
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are performed like a 1st embodiment. What is necessary is to replace with the image 
characteristic quantity Fi and positional attitude conversion Po which were calculated by 
a 1st embodiment, and just to use image-characteristic-quantity F'i and positional attitude 
conversion business sequence P'o at this time. 

[0082]As mentioned above, according to the robot device concerning a 2nd embodiment, 
even if the robot end point position P does not follow the subject 1 1 thoroughly, the 
subject 1 1 can be grasped only by performing easy teaching work mentioned above 
without carrying out complicated adjustment. 

[0083](A 3rd embodiment) According to 1st and 2nd embodiments, although the case of 
stillness movement and linear uniform motion was shown as an example, if this invention 
is used, it is expandable to a linear motion of acceleration or uniform circular motion. 
And processings from step ST1 1 to step ST 17 shown in drawing 7 are performed like [ 
these grip operations ] a 1st embodiment. 

[0084]In the case of a linear motion of acceleration, specifically the robot controller 5, In 
time ti to which the error vector E (t) became small enough, Judge with the robot end 
point position P having been relatively controlled by the position of Po or P'o to the 
subject 11, and V ac tuai (ti) in this time ti, Velocity component V es timate (t) generated from 
hand acceleration acc ac tuai 00, The speed commanding value which piled up the speed 
commanding value generated from Po or P'o is generated, the subject 1 1 is approached 
from the robot end point position P in time t, (ti), and the subject 1 1 is grasped by the 
gripper3. 

[0085]That is, the robot controller 5 is controlled to move the gripper 3 by Po from the 
robot end point position Ps when it changes into a flattery state to be shown in drawing 
14. The robot controller 5 computes P a according to the following formulas noting that it 
takes time T'o to carry out Po part relative displacement at this time. 
[0086] 
[Equation 6] 

Pa = PCO + Po+V^tj) -V+ /°V wtiMte (t)dt 
J 0 

[0087]Although the case where a subject was grasped was mentioned as the example and 
the embodiment mentioned above explained it, this invention is not limited to this. For 
example, even if it is a case where attachment of parts, etc. are carried out to the subject 
1 1 , it is applicable similarly. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing l] It is a figure showing the composition of the robot device concerning a 1st 
embodiment of this invention. 

[ Drawin g 2] It is an appearance perspective view of the subject grasped by a robot device. 
[Drawing 3 l it is a flow chart which shows the contents of processing in the teaching 
work of a robot device. 

[Drawing 4] In teaching work, it is a figure showing the rough composition which shows 
robot end point position Pd in the state where the subject was grasped. 
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[Drawing 5] In teaching work, it is a figure showing the rough composition which shows 
the robot end point position Ps when a subject is released. 

[Drawing 6] It is a figure showing the image pick of the camera in teaching work. 
[Drawing 7] It is a flow chart which shows the contents of processing in the grip 
operations of a robot device. 

[Drawing 8] In grip operations, it is a figure for explaining that the camera is picturizing 
the conveyor top which conveys a subject. 

[Drawing 9] It is a figure showing the image pick of the camera in grip operations. 
[Drawing 10] It is a figure for explaining the robot end point position P in time ti (ti), and 
speed commanding value vector V ac tuai (ti). 

[Drawing 11 "j it is a figure explaining a state when grasping a subject. 
[Drawing 12] It is a figure showing the picture of an initial state, a flattery state, and the 
subject at the time of re-instruction in the robot device concerning a 2nd embodiment. 
[Drawing 13] It is a figure explaining the speed commanding value vector V (t) of the 
robot end point position P. 

[Drawing 14] It is a figure explaining the state where the subject which the robot device 
concerning a 3rd embodiment of this invention is conveying was grasped. 
[Description of Notations] 

1 Camera 

2 Sensor controller 

3 Gripper 

4 Robot arm 

5 Robot controller 
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[Drawing 5] 
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[Drawing 7] 
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[Drawing 11] 




[Drawi ng 12 1 
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[ Drawing 14 ] 
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